VC-dimension Research Articles

Smoothed Analysis with Adaptive Adversaries

We prove novel algorithmic guarantees for several online problems in the smoothed analysis model. In this model, at each time step an adversary chooses an input distribution with density function bounded above pointwise by  \(\tfrac{1}{\sigma } \) times that of the uniform distribution; nature then samples an input from this distribution. Here, σ is a parameter that interpolates between the extremes of worst-case and average case analysis. Crucially, our results hold for adaptive adversaries that can base their choice of an input distribution on the decisions of the algorithm and the realizations of the inputs in the previous time steps. An adaptive adversary can nontrivially correlate inputs at different time steps with each other and with the algorithm’s current state; this appears to rule out the standard proof approaches in smoothed analysis. This paper presents a general technique for proving smoothed algorithmic guarantees against adaptive adversaries, in effect reducing the setting of an adaptive adversary to the much simpler case of an oblivious adversary (i.e., an adversary that commits in advance to the entire sequence of input distributions). We apply this technique to prove strong smoothed guarantees for three different problems: (1) Online learning: We consider the online prediction problem, where instances are generated from an adaptive sequence of σ -smooth distributions and the hypothesis class has VC dimension d . We bound the regret by \(\tilde{O}\big (\sqrt {T d\ln (1/\sigma)} + d\ln (T/\sigma) \big) \) and provide a near-matching lower bound. Our result shows that under smoothed analysis, learnability against adaptive adversaries is characterized by the finiteness of the VC dimension. This is as opposed to the worst-case analysis, where online learnability is characterized by Littlestone dimension (which is infinite even in the extremely restricted case of one-dimensional threshold functions). Our results fully answer an open question of Rakhlin et al. [64]. (2) Online discrepancy minimization: We consider the setting of the online Komlós problem, where the input is generated from an adaptive sequence of σ -smooth and isotropic distributions on the ℓ 2 unit ball. We bound the ℓ ∞ norm of the discrepancy vector by \(\tilde{O}\big (\ln ^2\big (\frac{nT}{\sigma }\big) \big) \) . This is as opposed to the worst-case analysis, where the tight discrepancy bound is \(\Theta (\sqrt {T/n}) \) . We show such polylog( nT / σ ) discrepancy guarantees are not achievable for non-isotropic σ -smooth distributions. (3) Dispersion in online optimization: We consider online optimization with piecewise Lipschitz functions where functions with ℓ discontinuities are chosen by a smoothed adaptive adversary and show that the resulting sequence is \(\big ({\sigma }/{\sqrt {T\ell }}, \tilde{O}\big (\sqrt {T\ell } \big)\big) \) -dispersed. That is, every ball of radius \({\sigma }/{\sqrt {T\ell }} \) is split by \(\tilde{O}\big (\sqrt {T\ell } \big) \) of the partitions made by these functions. This result matches the dispersion parameters of Balcan et al. [13] for oblivious smooth adversaries, up to logarithmic factors. On the other hand, worst-case sequences are trivially (0, T )-dispersed.

Understanding quantum machine learning also requires rethinking generalization

Quantum machine learning models have shown successful generalization performance even when trained with few data. In this work, through systematic randomization experiments, we show that traditional approaches to understanding generalization fail to explain the behavior of such quantum models. Our experiments reveal that state-of-the-art quantum neural networks accurately fit random states and random labeling of training data. This ability to memorize random data defies current notions of small generalization error, problematizing approaches that build on complexity measures such as the VC dimension, the Rademacher complexity, and all their uniform relatives. We complement our empirical results with a theoretical construction showing that quantum neural networks can fit arbitrary labels to quantum states, hinting at their memorization ability. Our results do not preclude the possibility of good generalization with few training data but rather rule out any possible guarantees based only on the properties of the model family. These findings expose a fundamental challenge in the conventional understanding of generalization in quantum machine learning and highlight the need for a paradigm shift in the study of quantum models for machine learning tasks.

Attention non-negative spectral clustering

Deep spectral clustering is an advanced unsupervised deep learning method with widespread applications. However, when applied to large-scale datasets, it may confront problems such as an unstable training process, limited scalability, and diminished interpretability. To mitigate these limitations, a novel unsupervised deep learning algorithm named Attention Non-negative Spectral Clustering (ANSC) is proposed. Specifically, a network architecture called the Spectral Attention Network (SAN) is designed to serve as the backbone network for ANSC. This design aims to improve the stability of the training process and the model’s scalability. This attention-based network architecture includes a feature extraction network, an eigenvector mapping network, and an orthogonalization module. Furthermore, to enhance the interpretability of ANSC, the objective function incorporates a cluster indicator matrix by introducing non-negative constraints. Additionally, the ℓ2,1-norm is employed as regularization to approximate the solution of the objective function. An analysis of a lower bound for the network size of ANSC is conducted based on the VC dimension. Extensive experiments validate that ANSC outperforms state-of-the-art clustering methods in terms of clustering accuracy, normalized mutual information, and adjusted rand index.

Parallelograms and the VC-dimension of the distance sets

In this paper, we study the distribution of parallelograms and rhombi in a given set in the plane over arbitrary finite fields Fq2. As an application, we improve the recent result due to Fitzpatrick, Iosevich, McDonald, and Wyman on the Vapnik–Chervonenkis dimension of the induced distance graph.

VC dimensions of group convolutional neural networks

We study the generalization capacity of group convolutional neural networks. We identify precise estimates for the VC dimensions of simple sets of group convolutional neural networks. In particular, we find that for infinite groups and appropriately chosen convolutional kernels, already two-parameter families of convolutional neural networks have an infinite VC dimension, despite being invariant to the action of an infinite group.

Compression schemes for concept classes induced by three types of discrete undirected graphical models

Sample compression schemes were first proposed by Littlestone and Warmuth in 1986. Undirected graphical model is a powerful tool for classification in statistical learning. In this paper, we consider labelled compression schemes for concept classes induced by discrete undirected graphical models. For the undirected graph of two vertices with no edge, where one vertex takes two values and the other vertex can take any finite number of values, we propose an algorithm to establish a labelled compression scheme of size VC dimension of associated concept class. Further, we extend the result to other two types of undirected graphical models and show the existence of labelled compression schemes of size VC dimension for induced concept classes. The work of this paper makes a step forward in solving sample compression problem for concept class induced by a general discrete undirected graphical model.

Boosting Simple Learners

Boosting is a celebrated machine learning approach which is based on the idea of combining weak and moderately inaccurate hypotheses to a strong and accurate one. We study boosting under the assumption that the weak hypotheses belong to a class of bounded capacity. This assumption is inspired by the common convention that weak hypotheses are "rules-of-thumbs" from an "easy-to-learn class". (Schapire and Freund~'12, Shalev-Shwartz and Ben-David '14.) Formally, we assume the class of weak hypotheses has a bounded VC dimension. We focus on two main questions: (i) Oracle Complexity: How many weak hypotheses are needed to produce an accurate hypothesis? We design a novel boosting algorithm and demonstrate that it circumvents a classical lower bound by Freund and Schapire ('95, '12). Whereas the lower bound shows that $\Omega({1}/{\gamma^2})$ weak hypotheses with $\gamma$-margin are sometimes necessary, our new method requires only $\tilde{O}({1}/{\gamma})$ weak hypothesis, provided that they belong to a class of bounded VC dimension. Unlike previous boosting algorithms which aggregate the weak hypotheses by majority votes, the new boosting algorithm uses more complex ("deeper") aggregation rules. We complement this result by showing that complex aggregation rules are in fact necessary to circumvent the aforementioned lower bound. (ii) Expressivity: Which tasks can be learned by boosting weak hypotheses from a bounded VC class? Can complex concepts that are "far away" from the class be learned? Towards answering the first question we {introduce combinatorial-geometric parameters which capture expressivity in boosting.} As a corollary we provide an affirmative answer to the second question for well-studied classes, including half-spaces and decision stumps. Along the way, we establish and exploit connections with Discrepancy Theory.

Lossy Compression With Universal Distortion

We consider a novel variant of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</i> -semifaithful lossy coding in which the distortion measure is revealed only to the encoder and only at run-time, as well as an extension of it in which the distortion constraint <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</i> is also revealed at run-time. Two forms of rate redundancy are used to analyze the performance, and achievability results of both a pointwise and minimax nature are demonstrated. The first coding scheme uses ideas from VC dimension and growth functions, the second uses appropriate quantization of the space of distortion measures, and the third relies on a random coding argument.

An Optimal Transport Analysis on Generalization in Deep Learning.

Deep neural networks (DNNs) have achieved state-of-the-art performance in various learning tasks, such as computer vision, natural language processing, and speech recognition. However, the fundamental theory of generalization still remains obscure in deep learning-why DNN models can generalize well, despite that they are heavily overparametrized in both depth and width? Recently, some work shows that traditional theory of analyzing the generalization error of learning models fails to explain the generalization of DNNs. The failure is mainly because of one simple fact that the worse case analysis of generalization error for learning models would be too loose for models with large parameter space, such as DNNs. In this work, we propose a new analysis of generalization in DNNs from an optimal transport perspective. Unlike traditional worse-case uniform convergence analysis in learning theory, our analysis of generalization error is dependent on both the learning algorithm and the data distribution and is the average-case analysis. Thus, our theory can be more practical and accurate to describe the generalization behavior of DNNs. More specifically, in this article, we try to answer a fundamental yet unsolved question in deep learning-why deeper models can generalize well than shallow models? The main contribution of this article can be summarized in four aspects. First, under a general learning framework, we derive upper bounds on the generalization error of learning algorithms by their algorithmic transport cost: the expected Wasserstein distance between the output hypothesis and the output hypothesis conditioned on an input example. We further provide several upper bounds on the algorithmic transport cost in terms of total variation distance, relative entropy, and Vapnik-Chervonenkis (VC) dimension. Moreover, we also study different conditions for loss functions under which the generalization error of a learning algorithm can be upper bounded by different probability metrics between distributions relating to the output hypothesis and/or the input data. Finally, under our established framework, we obtain our main results, showing that the generalization error in DNNs decreases exponentially to zero as the number of layers increases.

Induced odd cycle packing number, independent sets, and chromatic number

AbstractThe induced odd cycle packing number of a graph is the maximum integer such that contains an induced subgraph consisting of pairwise vertex‐disjoint odd cycles. Motivated by applications to geometric graphs, Bonamy et al. proved that graphs of bounded induced odd cycle packing number, bounded Vapnik–Chervonenkis (VC) dimension, and linear independence number admit a randomized efficient polynomial‐time approximation scheme for the independence number. We show that the assumption of bounded VC dimension is not necessary, exhibiting a randomized algorithm that for any integers and and any ‐vertex graph of induced odd cycle packing number at most returns in time an independent set of whose size is at least with high probability. In addition, we present ‐boundedness results for graphs with bounded odd cycle packing number, and use them to design a quasipolynomial‐time approximation scheme for the independence number only assuming bounded induced odd cycle packing number.

A Range Space with Constant VC Dimension for All-pairs Shortest Paths in Graphs

A Range Space with Constant VC Dimension for All-pairs Shortest Paths in Graphs

Minimum Excess Risk in Bayesian Learning

We analyze the best achievable performance of Bayesian learning under generative models by defining and upper-bounding the minimum excess risk (MER): the gap between the minimum expected loss attainable by learning from data and the minimum expected loss that could be achieved if the model realization were known. The definition of MER provides a principled way to define different notions of uncertainties in Bayesian learning, including the aleatoric uncertainty and the minimum epistemic uncertainty. Two methods for deriving upper bounds for the MER are presented. The first method, generally suitable for Bayesian learning with a parametric generative model, upper-bounds the MER by the conditional mutual information between the model parameters and the quantity being predicted given the observed data. It allows us to quantify the rate at which the MER decays to zero as more data becomes available. Under realizable models, this method also relates the MER to the richness of the generative function class, notably the VC dimension in binary classification. The second method, particularly suitable for Bayesian learning with a parametric predictive model, relates the MER to the minimum estimation error of the model parameters from data. It explicitly shows how the uncertainty in model parameter estimation translates to the MER and to the final prediction uncertainty. We also extend the definition and analysis of MER to the setting with multiple model families and the setting with nonparametric models. Along the discussions we draw some comparisons between the MER in Bayesian learning and the excess risk in frequentist learning.

ENSIGHTS: Intelligent Monitoring of Electric Power Transmission Assets

This work aims to use Data Science techniques to build predictive models that will eventually improve maintenance plans regarding power transformers by reducing shutoffs and transmission downtime. This work is part of a 36-month long Research and Development (R&amp;D) project started on January 2021, as of the writing of this report the project is halfway through. The analytic models described herein have already been tested while most of the remaining work is yet to be done. In a single Data Lake environment, we will consolidate several databases from information systems that support the company's operation and maintenance processes. Various machine learning models will run on this data, and their results will appear in a dashboard, alongside several traditional indicators. This process will be integrated and consolidated in a computing platform with cloud architecture. We use Machine Learning (ML) to develop the models. Based on an Agnostic Probably Approximately Correct (PAC) Learning study of the available datasets estimating their Vapnik–Chervonenkis dimension, we choose Random Forests (RF) algorithms to be used in the new indicators. So far, the project has produced two new indicators: Chromatographic Assay Indicator (CAI) and Electrical Failure Risk Indicator (EFRI). The CAI indicator evaluation uses a Random Forest Algorithm trained with an external dataset due to the small number of power transformers failures in the O&amp;M data. This indicator performed much better than classical chromatographic indicators to predict electric or temperature problems on the test set. The EFRI indicator correlates monitoring data available from an existing Supervisory Control and Data Acquisition (SCADA) system with maintenance data from an existing Enterprise Resource Planning (ERP) system through a RF algorithm capable of alerting to a higher risk of electric failure. ANEEL funds this work as R&amp;D project PD-00394-1907/2019 titled “Aplicabilidade de nova tecnologia voltada para o desenvolvimento de um modelo de monitoramento inteligente dos ativos de transmissão”.

General Vapnik–Chervonenkis dimension bounds for quantum circuit learning

Quantifying the model complexity of quantum circuits provides a guide to avoid overfitting in quantum machine learning. Previously we established a Vapnik–Chervonenkis (VC) dimension upper bound for ‘encoding-first’ quantum circuits, where the input layer is the first layer of the circuit. In this work, we prove a general VC dimension upper bound for quantum circuit learning including ‘data re-uploading’ circuits, where the input gates can be single qubit rotations anywhere in the circuit. A linear lower bound is also constructed. The properties of the bounds and approximation-estimation trade-off considerations are discussed.

On multivariate randomized classification trees: [formula omitted]-based sparsity, VC dimension and decomposition methods

Decision trees are widely-used classification and regression models because of their interpretability and good accuracy. Classical methods such as CART are based on greedy approaches but a growing attention has recently been devoted to optimal decision trees. We investigate the nonlinear continuous optimization formulation proposed in Blanquero et al. (2020) for training sparse optimal randomized classification trees. Sparsity is important not only for feature selection but also to improve interpretability. We first consider alternative methods to sparsify such trees based on concave approximations of the l0 “norm”. Promising results are obtained on 24 datasets in comparison with the original l1 and l∞ regularizations. Then, we derive bounds on the VC dimension of multivariate randomized classification trees. Finally, since training is computationally challenging for large datasets, we propose a general node-based decomposition scheme and a practical version of it. Experiments on larger datasets show that the proposed decomposition method is able to significantly reduce the training times without compromising the testing accuracy.

Diameter, Eccentricities and Distance Oracle Computations on H-Minor Free Graphs and Graphs of Bounded (Distance) Vapnik–Chervonenkis Dimension

Under the strong exponential-time hypothesis, the diameter of general unweighted graphs cannot be computed in truly subquadratic time (in the size of the input), as shown by Roditty and Williams. Nevertheless there are several graph classes for which this can be done such as bounded-treewidth graphs, interval graphs, and planar graphs, to name a few. We propose to study unweighted graphs of constant distance Vapnik–Chervonenkis (VC)-dimension as a broad generalization of many such classes—where the distance VC-dimension of a graph is defined as the VC-dimension of its ball hypergraph whose hyperedges are the balls of all possible radii and centers in . In particular for any fixed , the class of -minor free graphs has distance VC-dimension at most . Our first main result is a Monte Carlo algorithm that on graphs of distance VC-dimension at most , for any fixed , either computes the diameter or concludes that it is larger than in time , where only depends on and the notation suppresses polylogarithmic factors. We thus obtain a truly subquadratic-time parameterized algorithm for computing the diameter on such graphs. Then as a byproduct of our approach, we get a truly subquadratic-time randomized algorithm for constant diameter computation on all the nowhere dense graph classes. The latter classes include all proper minor-closed graph classes, bounded-degree graphs, and graphs of bounded expansion. Before our work, the only known such algorithm was resulting from an application of Courcelle’s theorem; see Grohe, Kreutzer, and Siebertz [J. ACM, 64 (2017), pp. 1–32]. For any graph of constant distance VC-dimension, we further prove the existence of an exact distance oracle in truly subquadratic space, that answers distance queries in truly sublinear time (in the number of vertices). The latter generalizes prior results on proper minor-closed graph classes to a much larger graph class. Finally, we show how to remove the dependency on for any graph class that excludes a fixed graph as a minor. More generally, our techniques apply to any graph with constant distance VC-dimension and polynomial expansion (or equivalently having strongly sublinear balanced separators). As a result for all such graphs one obtains a truly subquadratic-time deterministic algorithm for computing all the eccentricities, and thus both the diameter and the radius. Our approach can be generalized to the -minor free graphs with bounded positive integer weights. We note that all our algorithms for the diameter problem can be adapted for computing the radius, and more generally all the eccentricities. Our approach is based on the work of Chazelle and Welzl who proved the existence of spanning paths with strongly sublinear stabbing number for every hypergraph of constant VC-dimension. We show how to compute such paths efficiently by combining known algorithms for the stabbing number problem with a clever use of -nets, region decomposition, and other partition techniques.

Dot products in [formula omitted] and the Vapnik-Chervonenkis dimension

Given a set E⊂Fq3, where Fq is the field with q elements. Consider a set of “classifiers” Ht3(E)={hy:y∈E}, where hy(x)=1 if x⋅y=t, x∈E, and 0 otherwise. We are going to prove that if |E|≥Cq114, with a sufficiently large constant C>0, then the Vapnik-Chervonenkis dimension of Ht3(E) is equal to 3. In particular, this means that for sufficiently large subsets of Fq3, the Vapnik-Chervonenkis dimension of Ht3(E) is the same as the Vapnik-Chervonenkis dimension of Ht3(Fq3). In some sense the proof leads us to consider the most complicated possible configuration that can always be embedded in subsets of Fq3 of size ≥Cq114.

Disjointness through the Lens of Vapnik–Chervonenkis Dimension: Sparsity and Beyond

The disjointness problem—where Alice and Bob are given twosubsets of {1, dots, n} and they have to check if their setsintersect—is a central problem in the world of communicationcomplexity. While both deterministic and randomized communicationcomplexities for this problem are known to be Theta(n), it isalso known that if the sets are assumed to be drawn from somerestricted set systems then the communication complexity can be muchlower. In this work, we explore how communication complexitymeasures change with respect to the complexity of the underlying setsystem. The complexity measure for the set system that we use inthis work is the Vapnik—Chervonenkis (VC) dimension. Moreprecisely, on any set system with VC dimension bounded by d, weanalyze how large can the deterministic and randomized communicationcomplexities be, as a function of d and n. The d-sparse setdisjointness problem, where the sets have size at most d, is onesuch set system with VC dimension d. The deterministic and therandomized communication complexities of the d-sparse setdisjointness problem have been well studied and are known to beTheta left( d log left({n}/{d}right)right) and Theta(d),respectively, in the multi-round communication setting. In thispaper, we address the question of whether the randomizedcommunication complexity of the disjointness problem is always upperbounded by a function of the VC dimension of the set system, anddoes there always exist a gap between the deterministic andrandomized communication complexities of the disjointness problemfor set systems with small VC dimension.We construct two natural set systems of VC dimension d, motivated from geometry. Using these set systems, we show that the deterministic and randomized communication complexity can be widetilde{Theta}left(dlog left( n/d right)right) for set systems of VC dimension d and this matches the deterministic upper bound for all set systems of VC dimension d. We also study the deterministic and randomized communication complexities of the set intersection problem when sets belong to a set system of bounded VC dimension. We show that there exist set systems of VC dimension d such that both deterministic and randomized (one-way and multi-round) complexities for the set intersection problem can be as high as Thetaleft( dlog left( n/d right) right).

General Feasibility Bounds for Sample Average Approximation via Vapnik--Chervonenkis Dimension

We investigate the feasibility of sample average approximation (SAA) for general stochastic optimization problems, including two-stage stochastic programming without the relatively complete recourse assumption. Instead of analyzing problems with specific structures, we utilize results from the Vapnik-Chervonenkis (VC) dimension and Probably Approximately Correct learning to provide a general framework that offers explicit feasibility bounds for SAA solutions under minimal structural or distributional assumption. We show that, as long as the hypothesis class formed by the feasbible region has a finite VC dimension, the infeasibility of SAA solutions decreases exponentially with computable rates and explicitly identifiable accompanying constants. We demonstrate how our bounds apply more generally and competitively compared to existing results.

Concept learning consistency under three-way decision paradigm

Concept Mining is one of the main challenges both in Cognitive Computing and in Machine Learning. The ongoing improvement of solutions to address this issue raises the need to analyze whether the consistency of the learning process is preserved. This paper addresses a particular problem, namely, how the concept mining capability changes under the reconsideration of the hypothesis class. The issue will be raised from the point of view of the so-called Three-Way Decision (3WD) paradigm. The paradigm provides a sound framework to reconsider decision-making processes, including those assisted by Machine Learning. Thus, the paper aims to analyze the influence of 3WD techniques in the Concept Learning Process itself. For this purpose, we introduce new versions of the Vapnik-Chervonenkis dimension. Likewise, to illustrate how the formal approach can be instantiated in a particular model, the case of concept learning in (Fuzzy) Formal Concept Analysis is considered.